Adaptive user interface using machine learning mode

ABSTRACT

Techniques for social networking systems and methods for testing and applying user interfaces are disclosed herein. The method includes steps of presenting a user interface including a new user interface feature to a group of test users, collecting response data from the test users experiencing the user interface, performing analytics on the response data, and determining at least one interface rule of applying user interface features for a user depending on one or more user attributes of the user based on the analytics using a machine learning model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/612,554, entitled “ADAPTIVE USER INTERFACE USING MACHINE LEARNINGMODEL,” filed on Sep. 12, 2012, which is incorporated herein byreference in its entirety.

FIELD OF THE INVENTION

This invention relates generally to social networking, and in particularto methods and systems for testing and applying user interface featuresof a social networking system using machine learning models.

BACKGROUND

Modern social networking systems are interactive and support frequentuser inputs throughout the operation of the social networking system.There is a growing emphasis in social network development on userinterfaces designed to ease communication between the system and humans.

However, one drawback of existing social networking interfaces is thatthey have little ability to take into account differences in thepreferences, style, and knowledge of the users of the social networkingsystems. A social networking system can let a user select from a set ofdefault styles and even store a user's own interface variations, but thelatter process is manual and tedious. Clearly, there is a need forincreased personalization in the area of user interface for the socialnetworking system. The level of personalization needs to increase notonly in the types of flexibility but also in the way that thepersonalization occurs. Currently, most social networking systemsrequire that users determine or select their preferences explicitly tothe user interface, which means the options are either limited in numberor tiresome to complete. Moreover, some facets of user style or userpreference may be reflected in users' behavior but not subject toconscious inspection.

The suboptimal user interface of the existing social networking systemscan cause difficulty, distraction, or frustration for the users. As aresult, the social networking systems see lower conversion rate from theusers. For example, a user may find the registration process tedious andgive up. Or a user may ignore a friend message popped up on the upperright corner of the window since this user is used to focusing on theleft portion of the web page. Or a user may refuse to click anadvertisement link because the advertisement is bordered by a thickblack square block which has an unpleasant meaning in that user'sculture. To achieve better conversion rates, social networking systemsconstantly run user interface tests to experiment with new interfacefeatures. If a new feature wins an A/B test against an existing feature,the new feature will replace the existing feature in the user interface.However, these interface test procedures lack consideration of users'personal style and are not adaptive to the ever-changing trend of userpopulation inclination.

SUMMARY

The technology introduced here uses machine learning models to test andoptimize user interfaces of a social networking system, based on theobservation of user activity on the test user interfaces. The socialnetworking system presents new features for the user interfaces to agroup of test users and collects responses from the test users. Thesocial networking system performs analytics on the response data toassess how the new features works in view of groups of users who havedifferent user attributes (e.g. country, gender, age, etc.) from othergroups. The assessing is based on one or more metrics (e.g. a conversionrate) calculated from the user responses. Using Machine Learningalgorithms, the social networking system determines the markets (orother attributes) of which users the new feature or sequencing of flowworks best for.

The technology enables the social networking system to personalize theuser interfaces based on the empirical user data. The personalizationoccurs automatically without the intervention of the users. The systemcan even recognize subtle preferences reflected from users' behaviorsand determine user preferences accordingly. The technology optimizes theuser interfaces of the social networking system based on user groupswith different user attributes and maximizes the target metrics such asa conversion rate. Further the technology is adaptive to the trend ofuser group preferences since the machine learning model can periodicallyupdate the interface rule of applying user interface features.

The technology utilizes machine learning models to minimize the numberof interface rules and variations of the rules. The technology iscapable of determining an optimal set of the interface rules for anyuser based on the user attributes.

Other aspects of the technology introduced here will be apparent fromthe accompanying figures and from the detailed description whichfollows.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects, features and characteristics of the presentinvention will become more apparent to those skilled in the art from astudy of the following detailed description in conjunction with theappended claims and drawings, all of which form a part of thisspecification. In the drawings:

FIG. 1 illustrates a high level block diagram of a conventional processfor testing a user interface feature for a social networking system.

FIG. 2 illustrates a high level block diagram of a process for testing auser interface feature of a social networking system according to oneembodiment of the present invention.

FIG. 3 shows calculated metrics of registration success rates after ananalytic process.

FIGS. 4A-4D illustrate versions of a user interface for a social networkwith different features for testing.

FIG. 5 illustrates a high level block diagram of a process for testingthe user interfaces shown in FIGS. 4A-4D,

FIG. 6 shows calculated metrics of conversion rates after the analyticprocess in FIG. 5.

FIG. 7 illustrates a sample content of a database for storing interfacerules and attributes frequencies that a social networking systemmaintains.

FIG. 8 illustrates a sample content of a database for storing interfacerules and attributes-features correlation that a social networkingsystem maintains.

FIG. 9 illustrates a block diagram for inputs and outputs of aninterface testing process.

FIG. 10 is a high-level block diagram showing an example of thearchitecture of a node, which may represent any node in a cluster onwhich a social networking system described herein runs.

DETAILED DESCRIPTION

References in this specification to “an embodiment,” “one embodiment,”or the like, mean that the particular feature, structure, orcharacteristic being described is included in at least one embodiment ofthe present invention. All occurrences of such phrases in thisspecification do not necessarily refer to the same embodiment.

FIG. 1 illustrates a high level block diagram of a conventional processfor testing a user interface feature for a social networking system. Thesocial networking system runs A/B tests to test whether the new userinterface feature will perform better than the original feature forcertain goals. For example, the goal can be optimizing a conversion ratefor an advertisement link. In the example, the feature under the test isthe color of a button. The original color of the button is grey; the newuser interface contains a white button.

The social networking system can select a test group of test users totest the new feature of the white button. In addition, the socialnetworking system also selects a control group of control users toexperience the original user interface of the grey button. The testgroup and the control group of users are randomly selected to berepresentative of the same user population.

At 110, the social networking system rolls out the test user interfacewith the new feature, i.e. an advertisement link with a white button, tothe test group of test users. Then at step 112, the social networkingsystem collects responses from the test users. The responses includeinformation of whether the test users click the white button to accessthe advertisement. Similarly at 120, the social networking systempresents the original advertisement link with a grey button to thecontrol group of control users. At 122, the social networking systemcollects the responses from the control users, including information ofwhether the control users click the grey button to access theadvertisement.

Then at 130, the social networking system calculates the metrics of theconversion rate for the whole test group of test users and the wholecontrol group of control users, respectively. A comparison of thecalculated metrics for the test group and the control group leads to adecision 140 of whether to replace the original grey button with the newwhite button in the user interface for all users. Thus overall, the testimproves the metric of conversion rate for the entire user population.However, the interface test illustrated in FIG. 1 lacks consideration ofusers' personal styles and does not optimize the metric of conversionrate on a personal level.

FIG. 2 illustrates a high level block diagram of a process for testing auser interface feature of a social networking system, in accordance withone embodiment. The social networking system runs user interface teststo assess which variations on user interface features will perform bestfor certain goals. The user interface features can include, but are notlimited to, a property of a user interface element, a user interfaceelement, a sequence of interface flow, a user interface text, or acombination thereof. A feature is not necessarily a visual feature. Thegoal is to optimize or maximize a metric. The metric can be, but notlimited to, a conversion rate, a registration success rate, a timeperiod that a user spends on the user interface, an advertisement clickrate, a newsletter subscription rate, or a download rate. A conversionrate is a ratio of users who covert casual content views into desiredactions.

The goal of the process shown in FIG. 2 is to optimize the userregistration interface, in order to maximize the registration successrate, i.e. the percentage of the users loading the registration form whoactually click the submit button and finish the registration process.The feature under the test is the color of the submit button. Theoriginal color of the submit button is blue, and by contrast, the newuser interface contains a green submit button.

The social networking system randomly selects a test group of test usersto test the new feature of a green submit button. In addition, thesocial networking system randomly selects a control group of controlusers to experience the original user interface with a blue submitbutton. The test group and the control group of users are randomlyselected to be representative of the same user population. The sizes ofthe test group and the control group are large enough so that the testresult has a statistical significance and can be extrapolated to therest of the user population. In one embodiment, the sizes of the testgroup and the control group are substantially the same.

At 210, the social networking system rolls out the user registrationform with a green submit button (the new feature) to a test group oftest users. Then at step 212, the social networking system collectsresponses from the test users. The responses include information onwhether the test users click the green submit button to successfullysubmit the user registration form. Similarly at 220, the socialnetworking system presents the original user registration form with ablue submit button to the control group of control users. At 222, thesocial networking system collects the responses from the control users,including information on whether the control users click the blue submitbutton to successfully submit the original user registration form.

Although the metrics of a registration success rate can be calculatedfor the whole test group of test users and the whole control group ofcontrol users, such a comparison only leads to a decision of whether toreplace the original blue submit button with the new green submit buttonin the user interface for all users. Further analysis can be performedbased on the user attributes, leading to custom configured userexperience.

As shown in step 230 of FIG. 2, the social networking system conductsanalytics by calculating the metric F of a registration success rate fora plurality of subsets of users, each subset of users having differentuser attributes from other subsets. For example, the analytics can beconducted based on market (attribute a in FIG. 2) of the users andgender (attribute b in FIG. 2) of the users. For instance, a metric of aregistration success rate is calculated for a subset of test users whoare male and in US market and are experiencing the user interface with agreen submit button. Similarly, a metric of a registration success rateis calculated for a subset of control users who are male and in USmarket and are experiencing the user interface with a blue submitbutton. By comparing these two metrics, the social networking system candraw a statistical significant conclusion of whether male users in theUS market (a subset of users) prefer the green or blue submit button inthe registration form interface.

Likewise, the social networking system can conduct similar analytics onother user attributes, including but not limited to, country, region,number of friends, user activeness, location, race, age, user language,user device, browser, disability, home ownership, income, employment,education, or other user attributes that are statistically representableby the sizes of the test group and the control group. For instance,country or market attributes are commonly used in analytics becausesocial networks tend to be closely interconnected with a country or amarket.

Once the social networking system has conducted the analytics for usersubsets with different user attributes, the system contains knowledge ofwhether a subset of users who have particular user attributes prefersthe original user interface or the testing user interface with newfeature(s). At 240, using a machine learning model, based on theknowledge of analytics, the social networking system can determine aninterface rule of applying user interface features for a user dependingon one or more user attributes that are included for consideration inthe analytics. In some embodiments, a plurality of interface rules aredetermined for user subsets with different attributes.

For example, FIG. 3 shows the calculated metrics of registration successrates after the analytics process 230 in FIG. 2. For male users in theUS market (here assuming only two user attributes, market and gender,are considered by the machine learning model), by replacing the bluesubmit button on the registration form with the green submit button, theregistration success rate increases from 56% to 60%. For female users inthe US market, by replacing the blue submit button on the registrationform with the green submit button, the registration success rateincreases from 52% to 70%. For male users in the Canadian market, byreplacing the blue submit button on the registration form with the greensubmit button, the registration success rate decreases from 58% to 55%.For female users in the Canadian market, by replacing the blue submitbutton on the registration form with the green submit button, theregistration success rate decreases from 56% to 43%.

Based on the analyzed knowledge shown in FIG. 3, the social networkingsystem can automatically determining one or more rules of how to applyuser interface features depending on the user's attributes, using amachine learning model. For example, the social networking system canconclude from the analytics that users in the US market prefer the greensubmit button while users in the Canadian market prefer the blue submitbutton. The social networking system then can determine a rule that agreen submit button is applied to the registration form for users in theUS market: {market=US}=>{green submit button}, and another rule that ablue submit button is applied to the registration form for users in theCanadian market: {market=Canada}=>{blue submit button}.

In another embodiment, the social networking system can conclude fromthe analytics that male users do not have significant preferences on thecolors of the submit button. Rather than gaining a marginal increase ofthe registration success rate, the system may decide to keep the defaultcolor of the submit button for consistent user experience for maleusers. The social networking system then can determine a rule that thecolor of submit button in the registration form is applied with defaultvalues for the user's market for all male users: {gender=male}=>{defaultcolor submit button}. In addition, the social networking system canfurther determine a rule that a green submit button is applied to theregistration form for female users in the US market: {gender=female,market=US}=>{green submit button}, and another rule that a blue submitbutton is applied to the registration form for female users in theCanadian market: {gender=female, market=Canada}=>{blue submit button}.

These rules of applying user interface features are automaticallydetermined by the social networking system using a machine learningmodel, without human operator intervention. The sizes of the test groupand the control group are large enough so that the analytics results andrules are statistically significant for the entire user population thatthe test group and control group represent. These rules lead to a customconfigured experience for users having different user attributes (e.g.different markets). Thus, the social networking system personalizes theuser interface automatically based on the empirical user response data.

In one embodiment, the machine learning model automatically decideswhich user attribute makes statistical significance and is included indeciding the user interface. For instance, in FIG. 3, the machinelearning model may decide for all male users, the attribute of marketsis not significant attribute, since all male users in different marketsshow little interest in the color of the submit button. On the otherhand, the machine learning model may decide for all female users, theattribute of markets is a significant attribute, since markets play animportant role for female users on whether they prefer the green or bluesubmit button.

Similarly, the technology can be utilized in scenarios other than theregistration process. For example, a mobile application running on amobile device can benefit from a similar process to determine optimalGUI elements and sequences of flow for users of the mobile application.A website, including a social media website, can also use the technologydisclosed herein to determine an optimal set of interface rules appliedto its webpages based on user attributes.

The machine learning model can be any suitable model, such as decisiontree learning, association rule learning, artificial neural network,genetic programming, inductive logic programming, support vectormachines, clustering analysis, Bayesian network, reinforcement learning,representation learning, or sparse dictionary learning.

The technology can be further utilized to test multiple interfacefeatures in one test process. FIGS. 4A-4D illustrate versions of a userinterface for a social network with different features for testing, inaccordance with one embodiment. FIG. 4A illustrates a sample of anoriginal user interface for the social network. The original userinterface presents two questions, A and B, in two consecutive web pagesfor confirming a user agreement. Once a user clicks the square shape YESbutton on a first web page presenting the question A, the socialnetworking system presents the user with a second web page with thequestion B. After the user further clicks the square shape YES button onthe second web page, the user confirms that the user has read and agreedto comply with the user agreement.

FIG. 4B illustrates a sample of a user interface with a new interfacefeature. The new interface feature is a different sequence of interfaceflow. In FIG. 4B, the questions A and B are presented in a reversedorder. The first web page presents the question B to the user. Once theuser clicks the square shape YES button on a first web page presentingthe question B, the social networking system presents the user with asecond web page with the question A. After the user clicks the squareshape YES button on the second web page, the user confirms that the userhas read and agrees to comply with the user agreement.

FIG. 4C illustrates a sample of a user interface with a new interfacefeature. The new interface feature is an interface element having adifferent shape. In FIG. 4C, the YES button on both first and second webpages has an elliptical shape, instead of a square shape.

FIG. 4D illustrates a sample of a user interface with new interfacefeatures. The new interface features include a different sequence ofinterface flow and an interface element having a different shape. InFIG. 4D, the YES button on both first and second web pages has anelliptical shape, instead of a square shape. Further, the questions Aand B are presented in reversed order. The first web page presents thequestion B to the user. Once the user clicks the elliptical shape YESbutton on a first web page presenting the question B, the socialnetworking system presents the user with a second web page with thequestion A. After the user clicks the elliptical shape YES button on thesecond web page, the user confirms that the user has read and agrees tocomply with the user agreement.

FIG. 5 illustrates a high level block diagram of a process for testingthe user interfaces shown in FIGS. 4A-4D, in accordance with oneembodiment. The features under the test include the sequence ofinterface flow for presenting questions A and B, and the shape of theYES button for the user to click to confirm the user agreement. In orderto optimize the metric of conversion rate of users confirming the useragreement, the social networking system randomly selects users to formthree test groups of test users to test the new interfaces with newfeature(s). In addition, the social networking system randomly selectsusers to form a control group of control users to experience theoriginal user interface. The test groups and the control group of usersare randomly selected to be representative of the same user population.The sizes of the test groups and the control groups are large enough sothat the test result has a statistical significance to be extrapolatedto the rest of the user population.

At 510, the social networking system rolls out the new user interface 1having a reversed sequence of interface flow as shown in FIG. 4B to thefirst test group of test users. Then at step 512, the social networkingsystem collects responses from the test users. The responses includewhether the test users click the OK button on second web page to confirmthe user agreement. The social networking system also presents the newuser interface 2 having elliptical shape OK buttons as shown in FIG. 4Cto the second test group of test users at step 520. At step 522, thesocial networking system collects user responses, including whether thetest users click the OK button on the second web page to confirm theuser agreement.

The social networking system further presents the new user interface 3having elliptical shape OK buttons and reversed sequence of interfaceflow as shown in FIG. 4D to the third test group of test users at step530. At step 532, the social networking system collects user responses,including whether the test users click the OK button on the second webpage to confirm the user agreement. Similarly at 540, the socialnetworking system presents the original user interface as shown in FIG.4A to the control group of control users. At 542, the social networkingsystem collects the responses from the control users, including whetherthe test users click the OK button on the second web page to confirm theuser agreement.

As shown in step 550 of FIG. 5, the social networking system conductsanalytics by calculating the metrics F of the conversion rate for aplurality of subsets of users, each subset of users having differentuser attributes from other subsets. For example, the analytics can beconducted based on market (attribute a in FIG. 5), gender (attribute bin FIG. 5), and user device (attribute c in FIG. 5). For the purpose ofanalytics, the test groups and the control group are divided intosubsets of users having different attributes. For each user interfacethat is being tested, the metrics of conversion rate are calculated forthe subsets of users. Thus, the system contains knowledge of whether asubset of users having particular user attributes prefer the originaluser interface or a testing user interface with new feature(s).

The social networking system performs the division of user subsets insuch a way that small sample sizes of subsets are avoided. If the socialnetworking system identifies that different values of a particular userattribute can result in user subsets with such small sample sizes thatthe subsets lose statistical significance, the social networking systemrearranges the division of subsets for the user attribute (e.g.combining users having multiple attribute values into one subset), orabandons that user attribute from interface rule generation.

By analyzing the calculated metrics, the social networking systemselects attributes that have significant impacts on deciding the userinterface features using a machine learning model. Based on thecalculated metrics and selected attributes, at 560, the socialnetworking system can determine one or more rules of applying userinterface features for a user depending on one or more selected userattributes that are included for consideration in the analytics.

For example, FIG. 6 shows the calculated metrics of conversion ratesafter the analytics process 550 in FIG. 5. Using a machine learningmodel, the social networking system can deduct rules of applyingdifferent user interface to achieve customized interface for users, inorder to maximize the conversion rate. For example, the machine learningmodel may discover that female users in general, regardless of market oruser device, prefer the user interface with the elliptical submit buttonand [B, A] reversed order of sequence of flow. Accordingly, the socialnetworking system determines a rule of {gender=female}=>{elliptical,[B,A]}; i.e. for a user whose gender is female, a user interface forconfirming user agreement with elliptical submit button and [B, A]reversed order of sequence of flow is applied for the user, regardlessthe user's market or device.

Further, the machine learning model may discover that male users in theUS market, regardless of the user device being iPhone or Android, preferthe user interface with the square submit button and [A, B] order ofsequence of flow. Accordingly the social networking system determines arule of {gender=male, market=US}=>{square, [A,B]}; i.e. for a user whosegender is male and who is in US market, a user interface for confirminguser agreement with square submit button and [A, B] order of sequence offlow is applied for the user, regardless of the type of the user'sdevice.

By applying these automatically customized rules of applying userinterface, the conversion rate is maximized for each subset of users whohave different user attributes. The social networking systemautomatically conducts the analytics and rule generation using a machinelearning model without manual intervention from human operator. Theinputs that the social networking system needs are new user interfacefeatures to be tested and an indication of which metric to be optimized.Once the inputs are received, the social networking system selects testgroup(s) and control group accordingly. Then the social networkingsystem receives the user responses and calculating the target metricsbased on the user responses for each subset of users having differentattributes and for each test user interface. The social networkingsystem analyzes the calculated metrics and determines rules of applyingthe user interface features using the machine learning model.

Unlike a conventional A/B test, the output of the procedure is not onlya simple yes or no answer for whether to apply the new user interfacefeature for all users. Using a machine learning model, the socialnetworking system is able to conduct an intelligent procedure andoutputs rules about how to apply user interface features to differentsubsets of users, wherein each subset of users have different userattributes. In one embodiment, the output rules are human comprehensibleand statistically significant for the user population, and thereforeserve as useful guidelines for designing future user interface features.

The machine learning model automatically chooses which user attributemakes statistical significance on deciding the user interface features.For instance, based on the calculated metrics shown in FIG. 6, themachine learning model decides one rule based on the gender attributeand another rule based on the gender and market attributes, as discussedin previous paragraphs. In one embodiment, the social networking systemfurther monitors and records the frequencies of user attributesoccurring in the determined interface rules. The frequencies can be usedto help the machine learning model to select attributes in futureinterface tests. The user attributes that frequently show up ininterface rules, e.g. market attribute, can have higher priority thanless frequent attributes when the machine learning model choosesattributes for determining interface rules.

FIG. 7 illustrates a sample content of a database for interface rulesand attributes frequencies that a social networking system maintains, inaccordance with one embodiment. The database 700 records the interfacerules that are determined from interface tests. The social networkingsystem counts the frequencies of attributes occurring in the recordedinterface rules. Then, the database 700 further records the frequency ofeach attribute. In some embodiments, the frequencies of the userattributes may be used by the machine learning model when determiningthe order of attributes to be considered for interface rules.

In some embodiments, the database 700 can record the target metricsalong with the interface rules as well. For example, one interface rulecan be used to optimize the metric of registration success rate; anotherinterface rule can be used to optimize the metric of advertisementconversion rate. The social networking system can choose to applycertain interface rules for a certain metric to optimize the targetmetric.

Further, the social networking system may monitor and record not onlythe frequencies of the user attributes, but also the types of interfacefeatures that the user attributes are associated with in interfacerules. For example, FIG. 8 illustrates a sample content of a databasefor interface rules and attributes-features correlation that a socialnetworking system maintains, in accordance with one embodiment. Thedatabase 800 records the interface rules that are determined fromconducted interface tests. For each pair of a user attribute and aninterface feature, the social networking system counts the frequenciesof the attribute-interface pair occurring in the same existing interfacerule, and records the frequencies (as a correlation map) in the database800 as shown in FIG. 8.

The higher the frequency, the more closely a pair of a user attributeand an interface feature are correlated. When the social networkingsystem is conducting an interface test, the system identifies the typesof features being tested in the interface test. Then the socialnetworking system looks into its database to check if the types ofinterface features being tested have any close correlated userattributes. If the system finds any close correlated user attributes,the machine learning model will use these correlated user attributes inpriority for determining the interface rules for applying the correlatedfeatures being tested.

For example, as shown in the database 800 in FIG. 8, the interfacefeature of background color is closely correlated to the user attributegender. Further user attribute age is also correlated to the interfacefeature of background color. Therefore, when the social networkingsystem is conducting an interface test related to the choice ofbackground color, the machine learning model will first use theattributes of gender and age before other attributes for determining theinterface rules for applying the background color(s) being tested. Thisprocedure creates a bias toward the correlated attributes that arepredictive of optimizing the target metric (e.g. conversion rate).

Furthermore, the database 800 shows that the interface feature ofmessage font size is closely correlated to the user attribute of device.E.g. an Android users prefer bigger message font size than a PC user.When the social networking system is conducting an interface testrelated to the choice of message font size, the machine learning modelwill first use the attribute of device before other attributes fordetermining the interface rules for applying the message font size(s)being tested.

When the user attributes of a user are changed, manually by the user orautomatically by the system, the social networking system automaticallyapplies suitable user interface for the user, based on the new userattributes. For example, when a user travels to Canada and uses anAndroid cell phone designed for a Canadian cellular phone network toaccess a social network, the social networking system identifiesinterface rules in its database that match the new attributes (e.g.market=Canada, device=Android) and apply user interface for the useraccording to the identified interface rules.

The user attributes may have interdependencies upon each other. By usinga machine learning model, there is no need to remove interdependenciesby excluding user attributes from interface rules. The machine learningmodel can consider all available user attributes and automaticallygenerate rules for optimizing the target metric that is predetermined byan administrator or operator of the social networking system.

FIG. 9 illustrates a block diagram for inputs and outputs of aninterface testing process, in accordance with one embodiment. Anadministrator or operator 901 of the social networking system providesinputs that the social networking system needs for the interface test.The inputs include new user interface features 910 to be tested andinformation 920 about which metric to be optimized. The socialnetworking system 900 receives the inputs 910 and 920 and selects testgroup(s) and control group accordingly. The social networking system 900supplies test user interfaces 930 to users in test group(s) and controlgroup 940 and collects the user responses 950. The system calculates thetarget metrics based on the user responses for each subset of usershaving different attributes and for each tested user interface. Then,the social networking system analyzes the calculated metrics anddetermines interface rules of applying the user interface features usinga machine learning model. The social networking system outputs theinterface rules 960 and records the rules in its database 970. When auser 990 accesses the social network, the social networking system 900identifies and retrieves related interface rules from the database 970,and sends the user interface 980 according to the interface rules to theuser 990.

The social networking system can be implemented on a computer or acluster of computers (also referred to as nodes), such as a Hadoopcluster. The Hadoop cluster can include a plurality of nodes thatcommunicate with each other through an interconnect. The interconnectmay be, for example, a local area network (LAN), wide area network(WAN), metropolitan area network (MAN), global area network such as theInternet, a Fibre Channel fabric, or any combination of suchinterconnects. In some embodiments, the interconnect can include anetwork switch for processing and routing data between the nodes undernetwork protocols, including TCP/IP. Each of the nodes may be, forexample, a server, a conventional personal computer (PC), server-classcomputer, workstation, handheld computing/communication device, or thelike. In some embodiments, the cluster is implemented using one or moreracks of commodity-class servers.

FIG. 10 is a high-level block diagram showing an example of thearchitecture of a node, which may represent any node in a cluster onwhich a social networking system described herein runs. The node 1000includes one or more processors 1010 and memory 1020 coupled to aninterconnect 1030. The interconnect 1030 shown in FIG. 10 is anabstraction that represents any one or more separate physical buses,point to point connections, or both connected by appropriate bridges,adapters, or controllers. The interconnect 1030, therefore, may include,for example, a system bus, a Peripheral Component Interconnect (PCI) busor PCI-Express bus, a HyperTransport or industry standard architecture(ISA) bus, a small computer system interface (SCSI) bus, a universalserial bus (USB), IIC (I2C) bus, or an Institute of Electrical andElectronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The processor(s) 1010 is/are the central processing unit (CPU) of thestorage controller 1000 and, thus, control the overall operation of thenode 1000. In certain embodiments, the processor(s) 1010 accomplish thisby executing software or firmware stored in memory 1020. Theprocessor(s) 1010 may be, or may include, one or more programmablegeneral-purpose or special-purpose microprocessors, digital signalprocessors (DSPs), programmable controllers, application specificintegrated circuits (ASICs), programmable logic devices (PLDs), trustedplatform modules (TPMs), or the like, or a combination of such devices.

The memory 1020 is or includes the main memory of the node 1000. Thememory 1020 represents any form of random access memory (RAM), read-onlymemory (ROM), flash memory, or the like, or a combination of suchdevices. In use, the memory 1020 may contain, among other things, code1070 embodying at least a portion of procedures disclosed in previousparagraphs. Code 1070 may also include an analytic module stored in thememory that is executable by the processor to select a plurality ofsubsets of test users, wherein each of the subsets includes test usershaving different user attributes from other subsets of test users, andto calculate a metric for each of the test user interfaces and for eachof the subsets. Code 1070 may also include a machine learning modelmodule stored in the memory that is executable by the processor togenerate at least one interface rule associated with at least oneassociated user interface feature and at least one associated userattribute so that the metric for users who have the associated userattribute is optimized.

Also connected to the processor(s) 1010 through the interconnect 1030are a network adapter 1040 and a storage adapter 1050. The networkadapter 1040 provides the node 1000 with the ability to communicate withremote devices, over a network and may be, for example, an Ethernetadapter or Fibre Channel adapter. The network adapter 1040 may alsoprovide the node 1000 with the ability to communicate with other nodeswithin the cluster. In some embodiments, a node may use more than onenetwork adapter to deal with the communications within and outside ofthe cluster separately. The storage adapter 1050 allows the node 1000 toaccess a persistent storage, and may be, for example, a Fibre Channeladapter or SCSI adapter.

The code 1070 stored in memory 1020 may be implemented as softwareand/or firmware to program the processor(s) 1010 to carry out actionsdescribed above. In certain embodiments, such software or firmware maybe initially provided to the node 1000 by downloading it from a remotesystem through the node 1000 (e.g., via network adapter 1040).

The techniques introduced herein can be implemented by, for example,programmable circuitry (e.g., one or more microprocessors) programmedwith software and/or firmware, or entirely in special-purpose hardwiredcircuitry, or in a combination of such forms. Special-purpose hardwiredcircuitry may be in the form of, for example, one or moreapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), field-programmable gate arrays (FPGAs), etc.

Software or firmware for use in implementing the techniques introducedhere may be stored on a machine-readable storage medium and may beexecuted by one or more general-purpose or special-purpose programmablemicroprocessors. A “machine-readable storage medium”, as the term isused herein, includes any mechanism that can store information in a formaccessible by a machine (a machine may be, for example, a computer,network device, cellular phone, personal digital assistant (PDA),manufacturing tool, any device with one or more processors, etc.). Forexample, a machine-accessible storage medium includesrecordable/non-recordable media (e.g., read-only memory (ROM); randomaccess memory (RAM); magnetic disk storage media; optical storage media;flash memory devices; etc.), etc.

The term “logic”, as used herein, can include, for example, programmablecircuitry programmed with specific software and/or firmware,special-purpose hardwired circuitry, or a combination thereof.

In addition to the above mentioned examples, various other modificationsand alterations of the invention may be made without departing from theinvention. Accordingly, the above disclosure is not to be considered aslimiting and the appended claims are to be interpreted as encompassingthe true spirit and the entire scope of the invention.

What is claimed is:
 1. A computer implemented method comprising: storingone or more interface rules; receiving an indication of a metric valueto be optimized; presenting a user interface including a user interfacefeature to a group of users of a social networking system; collectingresponse data from the users experiencing the user interface;determining a plurality of attribute values for a user attribute of thegroup of users of the social networking system; for each attribute valueof the plurality of attribute values, calculating a value of the metricvalue for a subset of the group of users of the social networking systemwho have the user attribute of that attribute value, based on theresponse data; and analyzing using a machine learning model the valuesof the metric value for the subsets of the group of users of the socialnetworking system; identifying a list of frequently occurring userattributes by examining stored interface rules; generating at least oneinterface rule of applying the user interface feature depending on afrequently occurring user attribute from the list of the frequentlyoccurring user attributes based on the analysis of the machine learningmodel so that the metric value is optimized.
 2. The computer implementedmethod of claim 1, further comprising: generating a correlation mapbetween user interface features and user attributes by examining storedinterface rules; wherein the determining at least one interface rule ofapplying the user interface feature comprises: determining at least oneinterface rule of applying the user interface feature depending on theuser attribute based on an analysis using a machine learning model,wherein the user attribute is closely correlated to a user interfacefeature indicated in the interface rule according to the correlationmap.
 3. The computer implemented method of claim 2, wherein thecorrelation map records frequencies of pairs of user attributes andinterface features occurring in the stored interface rules, and whereineach frequency indicates a level of correlation between a user attributeand an interface feature of a corresponding fair.
 4. The computerimplemented method of claim 1, wherein the metric value is a conversionrate, a registration success rate, a time user spending on userinterface, an advertisement click rate, a newsletter subscription rate,or a download rate.
 5. The computer implemented method of claim 1,wherein the user interface features include at least one of userinterface element, sequence of interface flow, or user interface text.6. The computer implemented method of claim 1, further comprising:presenting an original user interface including an original userinterface feature to a group of control users; and collecting responsedata from the control users experiencing the original user interface. 7.The computer implemented method of claim 1, wherein the response dataincludes data of user inputs.
 8. The computer implemented method ofclaim 1, wherein the response data includes time which users spend onthe user interface.
 9. The computer implemented method of claim 1,further comprising: randomly selecting a plurality of users as a groupof test users.
 10. The computer implemented method of claim 1, whereinthe interface rule of applying user interface features is a decisionrule that is human comprehensible.
 11. The computer implemented methodof claim 1, wherein the group of users of the social networking systemhas such a size that the interface rule is statistically significant fora population of users.
 12. The computer implemented method of claim 1,wherein the presenting comprises: presenting a plurality of userinterfaces to a plurality groups of test users, each user interface ofthe plurality of user interfaces including at least one new userinterface feature.
 13. The computer implemented method of claim 1,further comprising: storing the interface rule; generating a userinterface according to the interface rule; and presenting the userinterface to a user, wherein the user is associated with a attributevalue that matches an attribute value associated with the interfacerule.
 14. A computer implemented method comprising: Storing one or moreinterface rules; receiving an indication of a metric value to beoptimized and a plurality of new user interface features to be tested;presenting a plurality of testing user interfaces to a plurality of testgroups of test users, wherein each testing user interface of theplurality of testing user interfaces include at least one of the newuser interface features; presenting an original user interface to acontrol group of control users, wherein the original user interface doesnot include any of the new user interface features; collecting responsedata from the test users experiencing the testing user interfaces andthe control users experiencing the original user interface; determininga plurality of attribute sets, each of attribute sets including one ormore user attributes; for each attribute set and each testing ororiginal user interfaces, calculating the metric value for a subset oftest or control users who have the user attributes of that attributeset, based on the response data; analyzing using a machine learningmodel the values of the metric value for the subsets of the group ofusers of the social networking system; identifying a list of frequentlyoccurring user attributes by examining stored interface rules;generating at least one interface rule associated with at least oneassociated user interface feature and at least one associated userattribute, depending on a frequently occurring user attribute from thelist of the frequently occurring user attributes based on the analysisof the machine learning model so that the metric value for users whohave the associated user attribute is optimized; and presenting a userinterface including the associated user interface to a user having theassociated user attribute, according to the interface rule.
 15. Thecomputer implemented method of claim 14, further comprising: generatinga correlation map between user interface features and user attributes byexamining stored interface rules; wherein the generating at least oneinterface rule comprises: generating at least one interface ruleassociated with at least one associated user interface feature and atleast one associated user attribute using a machine learning model sothat the metric value for users who have the associated user attributeis optimized, wherein the associated user attribute is closelycorrelated to the associated user interface feature indicated in theinterface rule according the correlation map.
 16. The computerimplemented method of claim 15, wherein the correlation map recordsfrequencies of pairs of user attributes and interface features occurringin the stored interface rules, and wherein each frequency indicates alevel of correlation between a user attribute and an interface featureof a corresponding fair.
 17. A system, comprising: a processor; a memoryfor storing one or more interface rules; an output interface forpresenting a plurality of test user interfaces to a plurality of groupsof test users, wherein each of the test user interfaces includes atleast one different user interface feature; an input interface forcollecting response data from the test users experiencing the test userinterfaces; an analytic module stored in the memory that is executableby the processor to select a plurality of subsets of test users, whereineach of the subsets includes test users having different user attributesfrom other subsets of test users, to calculate a metric value for eachof the test user interfaces and for each of the subsets; and a machinelearning model module stored in the memory that is executable by theprocessor to analyze the values of the metric value for the subsets ofthe group of users of the social networking system, to identify a listof frequently occurring user attributes by examining stored interfacerules; and to generate at least one interface rule associated to atleast one associated user interface feature and at least one associateduser attribute depending on at least one frequently occurring userattribute from the list of the frequently occurring user attributes sothat the metric value for users who have the associated user attributeis optimized.
 18. The system of claim 17, wherein the output interfaceis further for: presenting a user interface including the associateduser interface to a user has the associated user attribute, according tothe interface rule.
 19. The system of claim 17, further comprising: adatabase for storing existing interface rules and a list of frequentlyoccurring user attributes, wherein the list of frequently occurringattributes is generated by examining the existing interface rules storedin the database; and wherein the list of frequently occurring userattributes includes the associated user interface feature that isassociated by the interface rule determined by the machine learningmodel module.
 20. The system of claim 17, further comprising: a databasefor storing existing interface rules and a correlation map between userinterface features and user attributes, wherein the correlation map isgenerated by examining the existing interface rules stored in thedatabase; wherein the correlation map has an entry including theassociated user interface feature and the associated user attribute thatare associated by the interface rule determined by the machine learningmodel module.